{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 目录\n",
    "1. [分词算法](#一、分词算法)\n",
    "    1. [基于匹配规则的算法](#1、基于匹配规则的算法)\n",
    "    2. [基于概率统计的方法(LM,HMM,CRF..)](#2、基于概率统计的方法(LM,HMM,CRF..))\n",
    "        1. [维特比算法](#维特比(viterbi)算法:)\n",
    "        2. [HMM实现](#基于HMM模型的分词算法)\n",
    "2. [分词工具jieba](#二、分词工具jieba)    \n",
    "3. [TODO](#TODO：)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 一、分词算法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1、基于匹配规则的算法\n",
    "- 最大匹配算法(Max Matching)，按具体实现方式，分为前向、逆向、双向。\n",
    "- 给定词典 V，根据词典中单词长度的分布决定一个最大匹配长度，如 max_len=5，作为算法的参数。\n",
    "- 前向最大匹配算法示意图：\n",
    "    \n",
    "<img style=\"float: middle;\" src=\"../images/最大前向匹配.png\" width=\"60%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 逆向最大匹配算法实现\n",
    "- 时间复杂度：$O(n * max\\_len)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class InverseMaxMatching:\n",
    "    def __init__(self, dict_path):\n",
    "        self.dictionary = set()\n",
    "        self.max_length = 0\n",
    "        with open(dict_path, 'r', encoding='utf-8') as f:\n",
    "            for line in f:\n",
    "                line = line.strip()\n",
    "                if not line:\n",
    "                    continue\n",
    "                self.dictionary.add(line)\n",
    "                if len(line) > self.max_length:\n",
    "                    self.max_length = len(line)\n",
    "\n",
    "    def cut(self, text):\n",
    "        result = []\n",
    "        index = len(text)\n",
    "        while index > 0:\n",
    "            word = None\n",
    "            for size in range(self.max_length, 0, -1):\n",
    "                if index - size < 0:\n",
    "                    continue\n",
    "                piece = text[(index - size):index]\n",
    "                if piece in self.dictionary:\n",
    "                    word = piece\n",
    "                    result.append(word)\n",
    "                    index -= size\n",
    "                    break\n",
    "            if word is None:\n",
    "                index -= 1\n",
    "        return result[::-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "词典如下：\n",
      "深度\n",
      "学习\n",
      "深度学习\n",
      "很\n",
      "强大\n",
      "很强大\n"
     ]
    }
   ],
   "source": [
    "dict_path = 'datasets/imm_dic.utf8'\n",
    "print('词典如下：')\n",
    "with open(dict_path, 'r', encoding='utf-8') as f:\n",
    "    for line in f:\n",
    "        if line: print(line.strip())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['深度学习', '很强大']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = '深度学习很强大'\n",
    "tokenizer = InverseMaxMatching(dict_path)\n",
    "tokenizer.cut(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 匹配算法的缺点如下:\n",
    "- 只能得到局部最优\n",
    "- 效率低，时间复杂度为 $O(max\\_len * n)$\n",
    "- 不能细分\n",
    "- 不能考虑歧义"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2、基于概率统计的方法(LM,HMM,CRF..)\n",
    "- 给定**语言模型(LM)**，如 `Unigram-language model`，统计每个单词在语料库中出现的概率，如单词“经常”出现的概率：$P(经常) = \\frac{\"经常\"出现的次数}{语料库中单词总数}=\\frac{500}{10^6}$，其值通常为极小值。\n",
    "    \n",
    "- 生成句子所有可能的分割，如 s1= “经常/有/意见/分歧“，s2=“经常/有意见/分歧“ 两种分割方案，选择出现概率较大的那一种：\n",
    "   \n",
    "$$P(s1)=P(经常)\\cdot P(有)\\cdot P(意见)\\cdot P(分歧)$$    \n",
    "$$P(s2)=P(经常)\\cdot P(有意见)\\cdot P(分歧)$$\n",
    "- 因为每个单词的出现概率比较小，可能导致最终分割方案的概率极小，超出计算机处理范围(underfloat)，通常采用对数概率：\n",
    "$$logP(s1)=logP(经常)+logP(有)+logP(意见)+logP(分歧)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [维特比(`viterbi`)算法:](#目录)\n",
    "- 每个句子的分割方案可能很多，导致计算的复杂度很高；可以将其转化为求解最短路径的问题，利用维特比(`viterbi`)算法快速求解。\n",
    "$$f(m)=min\\bigg[f(i)+dist(i,m)\\bigg]\\quad{i\\in incomings\\ of\\ m}$$\n",
    "    \n",
    "- 示例：对 “经常有意见分歧” 进行分词\n",
    "\n",
    "词典：|经常|有|有意见|意见|分歧|见|见分歧|分|其它\n",
    "-|-|-|-|-|-|-|-|-|- \n",
    "概率：|0.1 |0.05|0.1| 0.2|0.2|0.05|0.05|0.1|2.1e-9   \n",
    " -log(x)：|2.3|3| 2.3| 1.6|1.6 |3 |3|2.3|20\n",
    "\n",
    "<img src=\"../images/维特比算法.png\" alt=\"维特比算法\" width=\"60%\"/>\n",
    "\n",
    "- 利用动态规划，可得出最短路径\n",
    "$$\n",
    "\\begin{align*}\n",
    "f(8)&=min\\{f(7)+dist(7,8);f(6)+dist(6,8);f(5)+dist(5,8)\\}\\\\\n",
    "f(7)&=min\\{f(6)+dist(6,7)\\}\\\\\n",
    "f(6)&=min\\{f(5)+dist(5,6);f(4)+dist(4,6);f(3)+dist(3,6)\\}\\\\\n",
    "f(5)&=min\\{f(4)+dist(4,5)\\}\\\\\n",
    "f(4)&=min\\{f(3)+dist(3,4)\\}\\\\\n",
    "f(3)&=min\\{f(2)+dist(2,3);f(1)+dist(1,3)\\}\\\\\n",
    "f(2)&=min\\{f(1)+dist(1,2)\\}\\\\\n",
    "f(1)&=0\\\\\n",
    "\\end{align*}\n",
    "$$\n",
    "- **分词可以认为是已经解决了的问题。**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [基于HMM模型的分词算法](#目录)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class HMM(object):\n",
    "    def __init__(self):\n",
    "        import os\n",
    "\n",
    "        # 主要是用于存取算法中间结果，不用每次都训练模型\n",
    "        self.model_file = 'models/hmm_model.pkl'\n",
    "\n",
    "        # 状态值集合\n",
    "        self.state_list = ['B', 'M', 'E', 'S']\n",
    "        # 参数加载,用于判断是否需要重新加载model_file\n",
    "        self.load_para = False\n",
    "\n",
    "    # 用于加载已计算的中间结果，当需要重新训练时，需初始化清空结果\n",
    "    def try_load_model(self, trained):\n",
    "        if trained:\n",
    "            import pickle\n",
    "            with open(self.model_file, 'rb') as f:\n",
    "                self.A_dic = pickle.load(f)\n",
    "                self.B_dic = pickle.load(f)\n",
    "                self.Pi_dic = pickle.load(f)\n",
    "                self.load_para = True\n",
    "\n",
    "        else:\n",
    "            # 状态转移概率（状态->状态的条件概率）\n",
    "            self.A_dic = {}\n",
    "            # 发射概率（状态->词语的条件概率）\n",
    "            self.B_dic = {}\n",
    "            # 状态的初始概率\n",
    "            self.Pi_dic = {}\n",
    "            self.load_para = False\n",
    "\n",
    "    # 计算转移概率、发射概率以及初始概率\n",
    "    def train(self, path):\n",
    "\n",
    "        # 重置几个概率矩阵\n",
    "        self.try_load_model(False)\n",
    "\n",
    "        # 统计状态出现次数，求p(o)\n",
    "        Count_dic = {}\n",
    "\n",
    "        # 初始化参数\n",
    "        def init_parameters():\n",
    "            for state in self.state_list:\n",
    "                self.A_dic[state] = {s: 0.0 for s in self.state_list}\n",
    "                self.Pi_dic[state] = 0.0\n",
    "                self.B_dic[state] = {}\n",
    "\n",
    "                Count_dic[state] = 0\n",
    "\n",
    "        def makeLabel(text):\n",
    "            out_text = []\n",
    "            if len(text) == 1:\n",
    "                out_text.append('S')\n",
    "            else:\n",
    "                out_text += ['B'] + ['M'] * (len(text) - 2) + ['E']\n",
    "\n",
    "            return out_text\n",
    "\n",
    "        init_parameters()\n",
    "        line_num = -1\n",
    "        # 观察者集合，主要是字以及标点等\n",
    "        words = set()\n",
    "        with open(path, encoding='utf8') as f:\n",
    "            for line in f:\n",
    "                line_num += 1\n",
    "\n",
    "                line = line.strip()\n",
    "                if not line:\n",
    "                    continue\n",
    "\n",
    "                word_list = [i for i in line if i != ' ']\n",
    "                words |= set(word_list)  # 更新字的集合\n",
    "\n",
    "                linelist = line.split()\n",
    "\n",
    "                line_state = []\n",
    "                for w in linelist:\n",
    "                    line_state.extend(makeLabel(w))\n",
    "\n",
    "                assert len(word_list) == len(line_state)\n",
    "\n",
    "                for k, v in enumerate(line_state):\n",
    "                    Count_dic[v] += 1\n",
    "                    if k == 0:\n",
    "                        self.Pi_dic[v] += 1  # 每个句子的第一个字的状态，用于计算初始状态概率\n",
    "                    else:\n",
    "                        self.A_dic[line_state[k - 1]][v] += 1  # 计算转移概率\n",
    "                        self.B_dic[line_state[k]][word_list[k]] = \\\n",
    "                            self.B_dic[line_state[k]].get(word_list[k], 0) + 1.0  # 计算发射概率\n",
    "\n",
    "        self.Pi_dic = {k: v * 1.0 / line_num for k, v in self.Pi_dic.items()}\n",
    "        self.A_dic = {\n",
    "            k: {k1: v1 / Count_dic[k]\n",
    "                for k1, v1 in v.items()}\n",
    "            for k, v in self.A_dic.items()\n",
    "        }\n",
    "        #加1平滑\n",
    "        self.B_dic = {\n",
    "            k: {k1: (v1 + 1) / Count_dic[k]\n",
    "                for k1, v1 in v.items()}\n",
    "            for k, v in self.B_dic.items()\n",
    "        }\n",
    "        #序列化\n",
    "        import pickle\n",
    "        with open(self.model_file, 'wb') as f:\n",
    "            pickle.dump(self.A_dic, f)\n",
    "            pickle.dump(self.B_dic, f)\n",
    "            pickle.dump(self.Pi_dic, f)\n",
    "\n",
    "        return self\n",
    "\n",
    "    def viterbi(self, text, states, start_p, trans_p, emit_p):\n",
    "        V = [{}]\n",
    "        path = {}\n",
    "        for y in states:\n",
    "            V[0][y] = start_p[y] * emit_p[y].get(text[0], 0)\n",
    "            path[y] = [y]\n",
    "        for t in range(1, len(text)):\n",
    "            V.append({})\n",
    "            newpath = {}\n",
    "\n",
    "            #检验训练的发射概率矩阵中是否有该字\n",
    "            neverSeen = text[t] not in emit_p['S'].keys() and \\\n",
    "                text[t] not in emit_p['M'].keys() and \\\n",
    "                text[t] not in emit_p['E'].keys() and \\\n",
    "                text[t] not in emit_p['B'].keys()\n",
    "            for y in states:\n",
    "                emitP = emit_p[y].get(text[t],\n",
    "                                      0) if not neverSeen else 1.0  #设置未知字单独成词\n",
    "                (prob, state) = max([\n",
    "                    (V[t - 1][y0] * trans_p[y0].get(y, 0) * emitP, y0)\n",
    "                    for y0 in states if V[t - 1][y0] > 0\n",
    "                ])\n",
    "                V[t][y] = prob\n",
    "                newpath[y] = path[state] + [y]\n",
    "            path = newpath\n",
    "\n",
    "        if emit_p['M'].get(text[-1], 0) > emit_p['S'].get(text[-1], 0):\n",
    "            (prob, state) = max([(V[len(text) - 1][y], y) for y in ('E', 'M')])\n",
    "        else:\n",
    "            (prob, state) = max([(V[len(text) - 1][y], y) for y in states])\n",
    "\n",
    "        return (prob, path[state])\n",
    "\n",
    "    def cut(self, text):\n",
    "        import os\n",
    "        if not self.load_para:\n",
    "            self.try_load_model(os.path.exists(self.model_file))\n",
    "        prob, pos_list = self.viterbi(text, self.state_list, self.Pi_dic,\n",
    "                                      self.A_dic, self.B_dic)\n",
    "        begin, next = 0, 0\n",
    "        for i, char in enumerate(text):\n",
    "            pos = pos_list[i]\n",
    "            if pos == 'B':\n",
    "                begin = i\n",
    "            elif pos == 'E':\n",
    "                yield text[begin:i + 1]\n",
    "                next = i + 1\n",
    "            elif pos == 'S':\n",
    "                yield char\n",
    "                next = i + 1\n",
    "        if next < len(text):\n",
    "            yield text[next:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<__main__.HMM at 0x7feb372f4a10>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hmm = HMM()\n",
    "hmm.train('datasets/trainCorpus.txt_utf8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "这是一个非常棒的方案！\n",
      "['这是', '一个', '非常', '棒', '的', '方案', '！']\n"
     ]
    }
   ],
   "source": [
    "text = '这是一个非常棒的方案！'\n",
    "res = hmm.cut(text)\n",
    "print(text)\n",
    "print(str(list(res)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [二、分词工具jieba](#目录)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**分词**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import jieba"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "全模式： 中文 分词 是 文本 文本处理 本处 处理 不可 不可或缺 或缺 的 一步 ！\n"
     ]
    }
   ],
   "source": [
    "sent = '中文分词是文本处理不可或缺的一步！'\n",
    "seg_list = jieba.cut(sent, cut_all=True)\n",
    "print('全模式：', ' ' .join(seg_list)) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "精确模式： 中文 分词 是 文本处理 不可或缺 的 一步 ！\n"
     ]
    }
   ],
   "source": [
    "seg_list = jieba.cut(sent, cut_all=False)\n",
    "print('精确模式：', ' '.join(seg_list)) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "默认精确模式： 中文 分词 是 文本处理 不可或缺 的 一步 ！\n"
     ]
    }
   ],
   "source": [
    "seg_list = jieba.cut(sent)  \n",
    "print('默认精确模式：', ' '.join(seg_list))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "搜索引擎模式： 中文 分词 是 文本 本处 处理 文本处理 不可 或缺 不可或缺 的 一步 ！\n"
     ]
    }
   ],
   "source": [
    "seg_list = jieba.cut_for_search(sent)  \n",
    "print('搜索引擎模式：', ' '.join(seg_list))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**带词性标注的分词**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "中文[nz]分词[n]是[v]文本处理[n]不可或缺[l]的[uj]一步[m]！[x]\n"
     ]
    }
   ],
   "source": [
    "import jieba.posseg as psg\n",
    "\n",
    "sent = '中文分词是文本处理不可或缺的一步！'\n",
    "seg_list = psg.cut(sent)\n",
    "\n",
    "print(''.join(['{0}[{1}]'.format(w, t) for w, t in seg_list]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**自定义词典**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "自定义词典内容：\n",
      "大波浪 10\n",
      "jieba分词 n\n",
      "金融词典 7\n"
     ]
    }
   ],
   "source": [
    "print('自定义词典内容：')\n",
    "with open('datasets/user_dict.utf8', 'r', encoding='utf-8') as f:\n",
    "    for line in f:\n",
    "        if line:\n",
    "            print(line.strip())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "系统词典: jieba 分词 非常 好用 ， 可以 自定义 金融 词典 ！\n"
     ]
    }
   ],
   "source": [
    "sent = 'jieba分词非常好用，可以自定义金融词典！'\n",
    "\n",
    "seg_list = jieba.cut(sent)\n",
    "print('系统词典:', ' '.join(seg_list))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "自定义词典: jieba分词 非常 好用 ， 可以 自定义 金融词典 ！\n"
     ]
    }
   ],
   "source": [
    "jieba.load_userdict('./datasets/user_dict.utf8')\n",
    "seg_list = jieba.cut(sent)\n",
    "print('自定义词典:', ' '.join(seg_list))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**调节特定词词频**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "好丑 的 证件 照片\n",
      "好丑 的 证件照片\n"
     ]
    }
   ],
   "source": [
    "sent = '好丑的证件照片'\n",
    "print(' '.join(jieba.cut(sent, HMM=False)))\n",
    "\n",
    "jieba.suggest_freq(('证件照片'), True)\n",
    "print(' '.join(jieba.cut(sent, HMM=False)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**提取关键词**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "自然语言 2.0790900005043476\n",
      "NLP 0.5197725001260869\n",
      "计算机 0.5197725001260869\n",
      "领域 0.5197725001260869\n",
      "人机交互 0.5197725001260869\n",
      "挑战 0.5197725001260869\n",
      "理解 0.5197725001260869\n",
      "处理 0.4705091875965217\n",
      "涉及 0.3839134341652174\n",
      "人工智能 0.25988625006304344\n"
     ]
    }
   ],
   "source": [
    "from jieba import analyse\n",
    "\n",
    "content = \"\"\"\n",
    "自然语言处理（NLP）是计算机科学，人工智能，语言学关注计算机和人类（自然）语言之间的相互作用的领域。\n",
    "因此，自然语言处理是与人机交互的领域有关的。在自然语言处理面临很多挑战，包括自然语言理解，因此，自然语言处理涉及人机交互的面积。\n",
    "在NLP诸多挑战涉及自然语言理解，即计算机源于人为或自然语言输入的意思，和其他涉及到自然语言生成。\n",
    "\"\"\"\n",
    "\n",
    "# 加载自定义idf词典\n",
    "analyse.set_idf_path('datasets/idf.txt.big')\n",
    "\n",
    "# 加载停用词典\n",
    "analyse.set_stop_words('datasets/stop_words.utf8')\n",
    "\n",
    "# 第一个参数：待提取关键词的文本\n",
    "# 第二个参数：返回关键词的数量，重要性从高到低排序\n",
    "# 第三个参数：是否同时返回每个关键词的权重\n",
    "# 第四个参数：词性过滤，为空表示不过滤，若提供则仅返回符合词性要求的关键词\n",
    "keywords = analyse.extract_tags(content, topK=10, withWeight=True, allowPOS=())\n",
    "\n",
    "for item in keywords:\n",
    "    print(item[0], item[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TextRank**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "涉及 1.0\n",
      "计算机 0.9618169519358478\n",
      "处理 0.8124660402732825\n",
      "理解 0.7885898958379202\n",
      "挑战 0.7833575495518058\n",
      "人机交互 0.7343470452632993\n",
      "语言学 0.727536034596871\n",
      "人类 0.6290562193534068\n",
      "人工智能 0.5809911385488661\n",
      "关注 0.577881611632419\n"
     ]
    }
   ],
   "source": [
    "# 第一个参数：待提取关键词的文本\n",
    "# 第二个参数：返回关键词的数量，重要性从高到低排序\n",
    "# 第三个参数：是否同时返回每个关键词的权重\n",
    "# 第四个参数：词性过滤，为空表示过滤所有，与TF—IDF不一样！\n",
    "keywords = jieba.analyse.textrank(content,\n",
    "                                  topK=10,\n",
    "                                  withWeight=True,\n",
    "                                  allowPOS=('ns', 'n', 'vn', 'v'))\n",
    "for item in keywords:\n",
    "    # 分别为关键词和相应的权重\n",
    "    print(item[0], item[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 处理文本文件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_content(path):\n",
    "    with open(path, 'r', encoding='gbk', errors='ignore') as f:\n",
    "        content = ''\n",
    "        for line in f:\n",
    "            line = line.strip()\n",
    "            content += line\n",
    "        return content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_tf(words, topK=10):\n",
    "    tf_dic = {}\n",
    "    for word in words:\n",
    "        tf_dic[word] = tf_dic.get(word, 0) + 1\n",
    "    return sorted(tf_dic.items(), key=lambda x: x[1], reverse=True)[:topK]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def stop_words(path):\n",
    "    with open(path, 'r', encoding='utf-8', errors='ignore') as f:\n",
    "        return [l.strip() for l in f]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "样本： 证券通：Ｇ 皖 通 车流量稳定增长 大股东可能注入优质资产（推荐）股票评级☆☆☆☆☆深度报告 权威内参 来自“证券通”www.KL178.comG皖通(行情,论坛)（600012）05年业绩增长迅速，计重收费方式下公司单车收入的大幅增长及合宁线的改造完成带来了利润的超速成长；在计重收费下的一次性惩罚性收入效应逐渐削弱的情况下，公司将通过内生车流量的增长，主营成本的下降（稳定的道路养护成本以及直线折旧法）的方式使得业绩在目前基础上依旧有望获得持续提升。考虑到公司车流量稳定增长以及大股东注入优质资产的可能，另外，根据股改协议未来数年高分红比率将能实现（未来数年股息收益率将在6％以上），参考A股公路定价水平，06年12-14倍PE定价是能够被市场接受的，合理价格区间是5.4-6.3元，目前股价存在一定低估，仍有14％－33％上升空间。因此给予投资评级为“推荐”。更多详情免费咨询021*64690729或登录www.KL178.com（证券通），资深行业研究员为您提供客观、深度、超前的投资信息。本文版权为“证券通”独家拥有，任何单位和个人不得复制、转发以及用于业务经营，违者将追究其法律责任。评级说明★★★★★ 坚决规避☆★★★★ 适当减仓☆☆★★★ 风险较大☆☆☆★★ 审慎推荐☆☆☆☆★ 首次推荐☆☆☆☆☆ 强烈推荐作者声明：在本机构、本人所知情的范围内,本机构、本人以及财产上的利害关系人与所述文章内容没有利害关系。本版文章纯属个人观点,仅供参考,文责自负。读者据此入市,风险自担。\n"
     ]
    }
   ],
   "source": [
    "import jieba\n",
    "import jieba.posseg as psg\n",
    "import random\n",
    "import glob\n",
    "\n",
    "files = glob.glob('datasets/news/C000008/*.txt')\n",
    "corpus = [get_content(path) for path in files[:5]]\n",
    "\n",
    "sample = random.choice(corpus)\n",
    "\n",
    "split_words = [\n",
    "    x for x in jieba.cut(sample)\n",
    "    if not x.isspace() and x not in stop_words('./datasets/stop_words.utf8')\n",
    "]\n",
    "print(\"样本：\", sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "分词结果： 证券|通|Ｇ|皖|通|车流量|稳定增长|股东|可能|注入|优质|资产|推荐|股票|评级|☆|☆|☆|☆|☆|深度|报告|权威|内参|证券|通|www|KL178|comG|皖通|行情|论坛|600012|05|年|业绩|增长|迅速|计重|收费|方式|公司|单车|收入|大幅|增长|及合宁线|改造|完成|带来|利润|超速|成长|计重|收费|一次性|惩罚性|收入|效应|逐渐|削弱|情况|公司|内生|车流量|增长|主营|成本|下降|稳定|道路|养护|成本|直线|折旧|法|方式|业绩|目前|基础|依旧|有望|获得|持续|提升|考虑|公司|车流量|稳定增长|股东|注入|优质|资产|可能|股改|协议|未来|数年|高|分红|比率|实现|未来|数年|股息|收益率|％|参考|A股|公路|定价|水平|06|年|12|14|倍|PE|定价|能够|市场|接受|合理|价格|区间|5.4|6.3|元|目前|股价|存在|一定|低估|14|％|－|33|％|上升|空间|给予|投资|评级|推荐|更|详情|免费|咨询|021|64690729|登录|www|KL178|com|证券|通|资深|行业|研究员|提供|客观|深度|超前|投资|信息|本文|版权|证券|通|独家|拥有|单位|个人|复制|转发|用于|业务|经营|违者|追究其|法律责任|评级|说明|★|★|★|★|★|坚决|规避|☆|★|★|★|★|适当|减仓|☆|☆|★|★|★|风险|较大|☆|☆|☆|★|★|审慎|推荐|☆|☆|☆|☆|★|首次|推荐|☆|☆|☆|☆|☆|强烈推荐|作者|声明|机构|知情|范围|机构|财产|利害|关系人|所述|文章内容|没有|利害关系|本版|文章|纯属|个人观点|仅供参考|文责自负|读者|入市|风险|自担\n"
     ]
    }
   ],
   "source": [
    "print(\"分词结果：\", '|'.join(split_words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "样本的topK(10)词： [('☆', 20), ('★', 15), ('通', 5), ('证券', 4), ('推荐', 4), ('车流量', 3), ('评级', 3), ('增长', 3), ('公司', 3), ('％', 3)]\n"
     ]
    }
   ],
   "source": [
    "print(\"样本的topK(10)词：\", str(get_tf(split_words)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "样本之一：证券通：Ｇ 皖 通 车流量稳定增长 大股东可能注入优质资产（推荐）股票评级☆☆☆☆☆深度报告 权威内参 来自“证券通”www.KL178.comG皖通(行情,论坛)（600012）05年业绩增长迅速，计重收费方式下公司单车收入的大幅增长及合宁线的改造完成带来了利润的超速成长；在计重收费下的一次性惩罚性收入效应逐渐削弱的情况下，公司将通过内生车流量的增长，主营成本的下降（稳定的道路养护成本以及直线折旧法）的方式使得业绩在目前基础上依旧有望获得持续提升。考虑到公司车流量稳定增长以及大股东注入优质资产的可能，另外，根据股改协议未来数年高分红比率将能实现（未来数年股息收益率将在6％以上），参考A股公路定价水平，06年12-14倍PE定价是能够被市场接受的，合理价格区间是5.4-6.3元，目前股价存在一定低估，仍有14％－33％上升空间。因此给予投资评级为“推荐”。更多详情免费咨询021*64690729或登录www.KL178.com（证券通），资深行业研究员为您提供客观、深度、超前的投资信息。本文版权为“证券通”独家拥有，任何单位和个人不得复制、转发以及用于业务经营，违者将追究其法律责任。评级说明★★★★★ 坚决规避☆★★★★ 适当减仓☆☆★★★ 风险较大☆☆☆★★ 审慎推荐☆☆☆☆★ 首次推荐☆☆☆☆☆ 强烈推荐作者声明：在本机构、本人所知情的范围内,本机构、本人以及财产上的利害关系人与所述文章内容没有利害关系。本版文章纯属个人观点,仅供参考,文责自负。读者据此入市,风险自担。\n"
     ]
    }
   ],
   "source": [
    "split_words = [\n",
    "    w for w, t in psg.cut(sample) if not w.isspace()\n",
    "    and w not in stop_words('datasets/stop_words.utf8') and t.startswith('n')\n",
    "]\n",
    "print('样本之一：' + sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "样本分词效果：证券/ 车流量/ 稳定增长/ 股东/ 优质/ 资产/ 股票/ 深度/ 报告/ 权威/ 内参/ 证券/ 皖通/ 行情/ 论坛/ 业绩/ 计重/ 收费/ 方式/ 公司/ 单车/ 合宁线/ 利润/ 计重/ 收费/ 惩罚性/ 效应/ 情况/ 公司/ 内生/ 车流量/ 成本/ 道路/ 成本/ 直线/ 方式/ 业绩/ 基础/ 公司/ 车流量/ 稳定增长/ 股东/ 优质/ 资产/ 协议/ 比率/ 股息/ 收益率/ A股/ 公路/ 定价/ 水平/ 定价/ 市场/ 价格/ 区间/ 股价/ 空间/ 证券/ 资深/ 行业/ 研究员/ 客观/ 深度/ 信息/ 版权/ 证券/ 独家/ 单位/ 个人/ 业务/ 违者/ 追究其/ 法律责任/ 风险/ 作者/ 声明/ 机构/ 知情/ 范围/ 机构/ 财产/ 关系人/ 文章内容/ 利害关系/ 文章/ 纯属/ 个人观点/ 读者/ 入市/ 风险\n"
     ]
    }
   ],
   "source": [
    "print('样本分词效果：' + '/ '.join(split_words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "样本的topK（10）词：[('证券', 4), ('车流量', 3), ('公司', 3), ('稳定增长', 2), ('股东', 2), ('优质', 2), ('资产', 2), ('深度', 2), ('业绩', 2), ('计重', 2)]\n"
     ]
    }
   ],
   "source": [
    "print('样本的topK（10）词：'+str(get_tf(split_words)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [TODO：](#目录)\n",
    "- 基于语言模型和维特比算法的分词实现\n",
    "- 基于 HMM 模型的理解"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
